Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: convert byte to string using unsafe, reduce memory alloc #19844

Merged
merged 4 commits into from
Nov 7, 2024

Conversation

huby2358
Copy link
Contributor

@huby2358 huby2358 commented Nov 7, 2024

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue # #18807

What this PR does / why we need it:

这里bytes转string存在大量内存申请,在profile alloc space里面占很大比重,而且这里是和load的文件大小成正比的,通过unsafe,强转,省去内存bytes转string存在的内存申请开销

@huby2358
Copy link
Contributor Author

huby2358 commented Nov 7, 2024

测试:深圳机器测试load sewer_001 parallel
时间性能前后对比:时间性能基本没有区别
修改前:
image

修改后:
企业微信截图_c5b32575-7887-4529-af99-0c10b060fb4d

heap alloc space:修改后,makeBtachRows总alloc space从64.8G降到18.9G,将str := string(parser.recordBuffer)这里占用的40多G省去了
修改前:
企业微信截图_c46f784a-6f51-42e4-87b1-952e3888c040

image 修改后: 企业微信截图_7c776524-4a20-4250-a912-a067a53e8db4

pkg/sql/util/csvparser/csv_parser.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/refactor Code refactor size/XS Denotes a PR that changes [1, 9] lines
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants